Validation in Distributed Representations

In the Hopfield model of content addressable memory, the number of spurious attractors is exponential in the dimensionality of the memory. Hence it is highly ’ikely that the system converges to a spurious memory on ar arbitrary input. It is desirable that the system has a way c:f checking whether its state corresponds to any one of the stored patterns. In this paper we show that it is possible to validate the patterns in a distributed fashion. The model uses complex activations and synapses for this purpose. The genuine memories are defined to be real attractors. Under some assumptions we show that the expected number of spurious attractors is negligibly small. We also calculate the capacity of the model. Oiie of the characteristic features of human memory is content addressabiiity. Based on partial input cues, the system retrieves one of the stored patterns. Implicit in this statement is the fact that the memory “knows” that tjhe retrieved item is something it has encountered before. People fail to remember but they seldom confuse unknown things with known things (unless they are hallucinating). Thus we need models of memory which, in addition to having content, addressability, can validate their input in light of their experience. The problem of validating the internal states of the system is called the validity problem. To make matters more concrete, consider the Hopfield model of content addressable memory[l]. Two important considerations of any neural model are: learning and computation. In this case these two correspond to storage and retrieval. In the Hapfield network, Hebb rule (also called the sum of outer products, in this case) is used for learning; relaxation is used as the mode of computation. During recall, the network iterates in its state space, {il}N, until convergence is achieved. The final state is taken as the recalled memory item. It is possible to show that the relaxation converges in all cases if the unih update themselves asynchronously. This can be proved by showing the existence of a function whose value decreases after each asynchronous update. ;Hence we can consider the stable states of the model as the local minima of this function. For this reason, the memories are also called the 6‘attractors’’. In the context of dynamical systems, it is desirable to distinguish between eyuilibrium, points and stable equilihrium points or attractors, An upper bound for the number of equilibrium points of the Bopfield network is N [2],[3]. What is desireble for a CAM is that the memories should also be stable equilibrium points. The niaximum number o f memories that can be stored as stable attractors i s called the capacity of the model. If Hebb rule is used for storage then the capacity of the Hopfield model is asymptotically & [4]. Another important performancemeiiure of a CAM is the number of spgrious memories (equilibrium points or attractors). The number of equilibrium points of the Hopfield network is exponential in the dimensionality [4][5]. Since at most N arbitrary patterns can be made equilibrium points of the model, the number of spurious equilibria is exponential in N . Hence hhere is a great likelihood that the network converges to a spurious attractor on an arbitrary input. The system has no way of checking whether the attractor corresponds to one of the stored memories. We need a mechanism to validate the state of the network in light of experience. In this paper we propose a mechanism to solve: this problem. The main idea is to make the network use complex numbers for its operation. In the complex domain, we define the genuine memories as real stable. equilibrium points. We can then expect that the number of spurious memories reduces greatly. In the rest of the paper we show that the above ideas indeed work.